Add list of non-evaluated artifacts based on ArtiFinder#147
Add list of non-evaluated artifacts based on ArtiFinder#147martonbognar wants to merge 12 commits into
Conversation
|
Could you host your branch in your fork? This way, it is easier to look at these changes. |
|
We are still working on refining the changes, but if you're interested, the current version is already deployed at https://mici.hu/secartifacts.github.io/ |
vahldiek
left a comment
vahldiek
left a comment
There was a problem hiding this comment.
left some comments. looks quite good already. Let me know what you think
| <td> | ||
| {% assign dir_name = page.path | remove_first:"_conferences/" | split:"/" | first %} | ||
| {% assign af_entry = site.data.artifinder_links[dir_name][include.title] %} | ||
| {% if af_entry %}<a href="{{ af_entry.url }}" target="_blank">Author's link</a>{% if af_entry.validated %} <abbr title="This extracted link has been manually validated." class="af-validated">✓</abbr>{% endif %}{% endif %} |
There was a problem hiding this comment.
Instead of calling it author link, could we create an artifact finder logo and use that instead? And have an alternative message that says something to the effect of extracted from the paper using artifact finder?
What does the manually validated mean here? every link has been clicked?
| The goal of artifact evaluation (AE) is to recognize the authors who have put in the effort to release | ||
| usable hardware and software systems as well as to validate the results of the accepted papers. | ||
|
|
||
| This website collects resources and results around artifact evaluation for security conferences and workshops. |
There was a problem hiding this comment.
I would add a sentence at the top here with a link to the artifacts without artifact evaluation section below.
|
|
||
| ## Artifacts without evaluation | ||
|
|
||
| While artifact evaluation is becoming more and more widespread, there are many papers that release artifacts without going through a formal evaluation process. |
There was a problem hiding this comment.
also historically speaking...
| While artifact evaluation is becoming more and more widespread, there are many papers that release artifacts without going through a formal evaluation process. | ||
| To increase the discoverability of these artifacts, the [ArtiFinder](https://github.com/DistriNet/ArtiFinder) tool was developed to automatically collect artifact URLs from papers. | ||
| The following pages list the automatically extracted artifacts. | ||
| As these results were not manually verified, mistakes are possible, for which pull requests are very welcome. |
There was a problem hiding this comment.
above I believe the text says manually verified, which is it?
| To increase the discoverability of these artifacts, the [ArtiFinder](https://github.com/DistriNet/ArtiFinder) tool was developed to automatically collect artifact URLs from papers. | ||
| The following pages list the automatically extracted artifacts. | ||
| As these results were not manually verified, mistakes are possible, for which pull requests are very welcome. | ||
| Currently, ArtiFinder was used to collect artifacts from A* conferences from the years 2000--2025. |
There was a problem hiding this comment.
I would rather name the conferences instead of saying A* - everyone has slightly different view of what A* means.
| @@ -0,0 +1,175 @@ | |||
| #!/usr/bin/env python3 | |||
There was a problem hiding this comment.
I would suggest to push these files into a separate folder maybe src or scripts or so...
There was a problem hiding this comment.
Do we need these script files or should we rather have the output files?
How would one submit an update to the artifact URL? Through the artfinder repo?
There was a problem hiding this comment.
Yes, this was one of the main technical aspects I was wondering about. I would like to keep the dataset in a central location and not duplicate it, so it can serve as a single source of truth, and I'm not sure the secartifacts website repo is the best place for that. This also means that corrections/validations should also be sent to our dataset repository. Whether it's included as a submodule or just fetched dynamically when building the website doesn't matter too much I guess, for now a submodule felt cleaner. Initially I also didn't want these additional generated files, but without them the website generation took excruciatingly long.
Regarding the process of submitting contributions: we're looking into simplifying this process as much as possible, e.g., GitHub now supports issue templates, which might be a good solution...
There was a problem hiding this comment.
that makes sense, is there still a path to keep the scripts in the artfinder repo and only have the deployment change (the workflow changes that you have right now)? Otherwise I would at least suggest to put these files into their own folder in this repo...
Suggestion on template issues is fine.
| Available At | ||
| </th> | ||
| <th> | ||
| Author's link |
There was a problem hiding this comment.
I wouldn't call those author's link. Isn't this artfinder/paper link?
|
Artfinder is limited by public papers I guess. Any chance to run it through systems conferences as well or are they behind ACM DL paywall? |
|
Something that I just realize is that because the data is not generated into the yaml files. reproDB will have to directly download the artfinder output files from the repo. If you have any suggestion, please let me know. I was thinking to add those before July 20th... |
2 participants
This pull request (currently a placeholder) serves to add the list of artifacts we obtained using ArtiFinder, a tool to automatically identify research artifacts from papers. We built a dataset that contains artifacts published at IEEE S&P, ACM CCS, USENIX Security, and NDSS in the period 2000--2025, and ACSAC in 2017--2025.
The current proposal adds the following content: